To facilitate new video applications, such as three-dimensional television (3DTV) and free-viewpoint video (FVV), 3D Video (3DV) data formats comprising both conventional 2D video and depth maps can be utilized such that additional views can be rendered at the user end. Examples of such 3DV formats include 2D plus depth (2D+Z), which includes a two-dimensional (2D) video and its corresponding depth map, and Layered Depth Video (LDV), which includes 2D+Z and an occlusion video plus an occlusion depth. Other examples of such 3DV formats include Multiview plus Depth (MVD) and Disparity Enhanced Stereo (DES). MVD is an extension of 2D+Z, as it includes multiple 2D+Z from different viewpoints. In turn, DES is composed of two LDVs from two different view points. Another example 3DV format is Layer Depth Video plus Right View (LDV+R) which is composed of one LDV of a left view and the 2D video of the right view. How to convey (encode and transmit) the data in all these formats is a challenging issue, as different components are used jointly at the user end to decode 3DV content.